CONCURRENT DUAL PIPELINE FOR ACQUISITION, PROCESSING 
AND TRANSMISSION OF DIGITAL VIDEO AND HIGH RESOLUTION 
DIGITAL STILL PHOTOGRAPHS 

Technical Field 

The technical field relates to video imaging systems, and, in particular, to 
concurrent dual video and still image pipelines. 
Background 

Digital cameras are widely used to acquire high resolution still image 
photographs. Digital video cameras are also used to record home videos, television 
programs, movies, concerts, or sports events on a disk or DVD for storage or transmission 
through communications channels. Some commercial cameras are able to take both 
digital video and digital still image photographs. However, most of these cameras 
required a user to switch between a video recording mode and a digital still image mode. 
Separate pipelines are generally used for each of the video recording and still image 
modes. Examples of these cameras include SANYO ID-SHOT® and CANNON 
POWERSHOT S300®. The SANYO ID-SHOT® uses an optical disk, whereas the 
CANNON POWERSHOT S3 00® uses one of the synchronous dynamic random access 
memory (SDRAM). However, both products are still image cameras that have the 
capability of taking video clips, using separate pipelines. 

Other cameras use a single software pipeline to acquire both digital video and low 
quality still images, by taking one of the video frames as is, and storing the particular 
video frame as a still image. Examples of such cameras include JVC GR-DVL9800®, 
which is a digital video camera that allows a user to take a picture at certain point in time. 
However, the pictures taken generally are of low quality, because a low resolution video 
pipeline is used to generate the still image pictures. 
Summary 

A method for concurrently acquiring, processing, and transmitting digital video 
and still images includes acquiring video frames from one or more image sensors, and 
processing the video frames using a video pipeline. The video pipeline may include one 
or more processors. The method further includes temporarily storing the video frames in 
a frame buffer when one or more high resolution still images are acquired during the 
video frame acquisition, and processing the high resolution still images using a still image 
pipeline. The still image pipeline may run concurrently with the video pipeline. 
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1 As a result, high resolution still image photographs may be acquired during video 

2 recording without any interference with the video recording or any impact on the quality 

3 of the video or the still image photographs. 

4 Description of the Drawings 

5 The preferred embodiments of the concurrent dual video and still image pipeline 

6 will be described in detail with reference to the following figures, in which like numerals 

7 refer to like elements, and wherein: 

8 Figure 1 illustrates an exemplary operation of an exemplary concurrent dual video 

9 and still image pipeline; 

1 0 Figure 2 illustrates a preferred embodiment of a video camera system using the 

1 1 exemplary concurrent dual video and still image pipeline of Figure 1 ; 

12 Figure 3 illustrates an exemplary hardware implementation of the exemplary 
W 1 3 concurrent dual video and still image pipeline of Figure 1 ; 

q 14 Figures 4 A- 4C are flow charts describing in general the exemplary concurrent 

^ 1 5 dual video and still image pipeline of Figure 1 ; and 

m 16 Figure 5 illustrates an alternative embodiment of the exemplary concurrent dual 

fill 

pi 17 video and still image pipeline of Figure 1 . 

I s * 1 8 Detailed Description 

Q 

m 19 A digital video camera system may utilize a concurrent dual video and still image 

20 pipeline that simultaneously acquires, processes^ transmits and/or stores digital video and 

21 high resolution digital still image photographs. The concurrent dual pipeline may include 

22 a video pipeline optimized for digital video and a still image pipeline optimized for high 

23 resolution digital still images. As a result, high resolution still image photographs may be 

24 acquired during video recording without any interference with the video recording or any 

25 impact on the quality of the video or the still image photographs. 

26 Figure 1 illustrates an exemplary operation of an exemplary concurrent dual video 

27 and still image pipeline, which is capable of simultaneously delivering digital video 

28 frames 120 and high resolution digital still images 110. The video frames 120 may be 

29 acquired at, for example, 30 frames per second (fps). During video frame acquisition, a 

30 - snapshot 102 may be taken to acquire a particular still image 1 10 in high resolution. 

31 During the high resolution still image acquisition, all other video frames 120 may be 

32 temporarily stored, i.e., buffered, in a frame buffer 330 (shown in Figure 3). Both the 

33 video frames 120 and the high resolution still image 110 may be transmitted through 

34 communications channels, such as a network. 
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Figure 2 illustrates a preferred embodiment of a video camera system 200 using 
the exemplary concurrent dual video and still image pipeline. In this embodiment, a 
video pipeline 220 and a still image pipeline 210 share a same high resolution image 
sensor 240. The high resolution image sensor 240, which may be a charge coupled 
device (CCD) sensor or a complimentary metal oxide semiconductor (COMS) sensor, 
may need to take high resolution still images 110 while acquiring medium resolution 
video frames 120. This embodiment is relatively inexpensive, because the video camera 
system 200 utilizes only one hardware processing pipeline 300 (shown in Figure 3) with 
one image sensor 240 and one processor 360 (shown in Figure 3). 

The image sensor 240 typically continuously acquires high resolution video 
frames 120 at a rate of, for example, 30 fps in the United States. Each of the high 
resolution video frames 1 20 may possess enough information to generate a high 
resolution still image photograph 110. When a user is not interested in taking a high 
resolution still image photograph 110, the only pipeline that is running may be the 
medium resolution (for example, 720x480) video pipeline 220. When the user wants to 
acquire a high resolution still image 1 10, the image acquired by the high resolution image 
sensor 240 at that particular time may be used both in the video pipeline 220 as well as in 
the high resolution still image pipeline 210 (described in detail later). 

The video camera system 200 may include a storage device 250 and a connection 
with a communications channel/network 260, such as the Internet or other type of 
computer or telephone networks. The storage device 250 may include a hard disk drive, 
floppy disk drive, CD-ROM drive, or other types of non- volatile data storage, and may 
correspond with various databases or other resources. After the video frames 120 and the 
high resolution still images 110 are acquired, the video frames 120 and the high 
resolution still images 110 may be stored in the storage device 250 or transmitted through 
the communication channel 260. 

Figure 3 illustrates an exemplary hardware implementation of the preferred 
embodiment of the exemplary concurrent dual video and still image pipeline. This 
embodiment includes the single hardware processing pipeline 300 supporting two 
software pipelines. A sensor controller 3.1.0 may be controlled by a user to retrieve high 
resolution mosaiced still images 1 10 at a rate of, for example, one every thirtieth of a 
second to generate a video signal. The sensor controller 3 1 0 may then store the selected 
high resolution still images 110 into a memory 320. The memory 320 may include 
random access memory (RAM) or similar types of memory. Next, the high resolution 
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still images 1 10 may be processed using the processor 360, which may be a 
microprocessor 362, an ASIC 364, or a digital signal processor 366. The ASIC 362 
performs algorithms quickly, but is typically application specific and only performs a 
specific algorithm. On the other hand, the microprocessor 362 or the digital signal 
processor 366 may perform many other tasks. The processor 360 may execute 
information stored in the memory 320 or the storage device 250, or information received 
from the Internet or other network 260. The digital video and still image data may be 
copied to various components of the pipeline 300 over a data bus 370. 

In the video pipeline 220, the processor 360 may downsample, demosaic, and 
color correct the video frames 120. Next, the processor 360 may compress and transmit 
the video frames 120 through an input/output (I/O) unit 340. Alternatively, the video 
frames 120 may be stored in the storage device 250. 

Both pipelines may be executed concurrently, i.e., acquiring high resolution still 
image photographs 110 during video recording. A frame buffer 330 may store video 
frames 120 while the single processor 360 is processing the high resolution still image 
110. The sensor controller 3 1 0 may still capture video frames 120 at a rate of, for 
example, 30 fps, and store the video frames 120 into the memory 320. The processor 360 
may need to downsample the video frames 120 and send the downsampled video frames 
120 into the frame buffer 330. The frame buffer 330 may store the downsampled video 
frames 120 temporarily without further processing, incurring some delay in the video 
pipeline 220. However, this delay may be compensated by a similar buffer on the 
receiver end (described later). During video frame buffering, the high resolution still 
image 110 may be processed by the processor 360, using complex algorithms. At the 
same time, the video frames 120 are continuously stored into the memory 320, 
downsampled, and sent into the frame buffer 330 to be stored. 

During the time the processor 360 processes the high resolution still image 1 10, 
the frame buffer 330 is being filled. Once the high resolution still image 1 10 has been 
processed, transmitted and/or stored, the video pipeline 220 takes over and starts 
processing the video frames 120 stored and accumulated in the frame buffer 330. 
Thereafter, the video pipeline 220 transmits the video frames 120 into the I/O unit 340 or 
the storage device 250 in order to empty the frame buffer 330. At the same time, new 
video frames 120 may be acquired and stored into the memory 320. Accordingly, the 
video pipeline 220 may need to clear the frame buffer 330 faster than the speed the video 
frames 120 are acquired. After the frame buffer 330 is emptied, another high resolution 
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still image 110 may be taken. The frequency of high resolution still image 110 that can 
be acquired depends on the capacity of the processor 360 and the size of the frame buffer 
330. 

If the video frames 120 are stored, the above operation may not affect the overall 
video camera system 200, since a delay in storing may not be noticeable. On the other 
hand, if the video frames 120 are transmitted, the delay may be noticed. However, all 
video streaming solutions have another frame buffer on the receiver end, which may 
absorb the delay, so the viewer at the other end of the communications channel may not 
notice the delay. 

Alternatively, the high resolution still image 1 10 may be acquired and stored 
without any processing. During this procedure, some video frames 120 may be stored in 
the frame buffer 330 to move the large amount of unprocessed pixels across the data bus 
370 into the storage device 250. In this embodiment, the frame buffer 330 may be much 
smaller, and the delay in video transmission may be smaller. However, the high 
resolution still images 110 may not be available until the video acquisition stops and 
allows the video camera system 200 to process all the high resolution still images 110 that 
have been acquired during the video recording. Accordingly, any image transmission that 
the user desires may be delayed. The above described solutions are for illustration only, 
and one skilled in the art will appreciate that many other intermediate solutions will be 
possible with this architecture for the concurrent dual video and still image pipeline. 

Although the video camera system 200 is depicted with various components, one 
skilled in the art will appreciate that the video camera system 200 can contain additional 
or different components. In addition, although aspects of an implementation consistent 
with the present invention are described as being stored in memory, one skilled in the art 
will appreciate that these aspects can also be stored on or read from other types of 
computer program products or computer-readable media, such as secondary storage 
devices, including hard disks, floppy disks, or CD-ROM; a carrier wave from the Internet 
or other network; or other forms of RAM or ROM. The computer-readable media may 
include instructions for controlling the video camera system 200 to perform a particular 
method. 

Figures 4A- 4C are flow charts describing in general the exemplary concurrent 
dual video and still image pipeline. Referring to Figure 4A, operation of the video 
pipeline 220, shown on the left, typically results in continuous processing of video frames 
120. Operation of the still image pipeline 210, shown on the right, typically results in 
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processing a high resolution still image 110 every time the user wants to acquire a high 
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resolution photograph 110. 
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After raw pixel video data of video frames 1 20 are acquired, for example, at 
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demosaiced in order to save memory space (block 410). Then, the Irame butter 330 may 
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buffer the video frames 120 while the high resolution still image 1 10 is being acquired, 
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processed, stored, and/or transmitted (block 420). Alternatively, demosaicing may be 
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performed after the video frames 120 are buffered. Thereafter, the video pipeline 220 
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may start emptying the frame buffer 330 as fast as possible. Once the frame buffer 330 is 
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emptied, another high resolution still image 1 10 may be acquired. Next, color correction 
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may be performed (block 430), followed by compression of the video frames 120 (block 
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(block 412), followed by high quality color correction (block 432). The high resolution 




i & 
lo 


still images 110 may optionally be compressed (block 442), and then stored and/or 


3 Jl 


1 *7 

17 


transmitted through similar communications channels 260 (block 452). 


rts 


1 o 

18 


Figure 4B illustrates in detail the operation of the high resolution still image 
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pipeline 210. The sophisticated demosaicing process (block 412) utilizes a high quality 
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demosaicing algorithm that generates a high quality color image from the originally 
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mosaiced image acquired by the image sensor 240. Examples or the demosaicing 
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SCALE INVARIANT PRIOR by David Taubman, Proceedings of the ICIP2000, » 
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published September 10, 2000, which is incorporated herein by reference. The 
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demosaicing process is a time consuming filtering operation, which may gamma-correct 
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the input if the image sensor 240 has not done so, resulting in excellent color image 
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quality with almost no demosaicing artifacts. For example, demosaicing for high 
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resolution still images 110 may filter the original image with a 10x10 linear filter. The 
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algorithm takes into account the lens used for acquisition, as well as the spectral 
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sensitivity of each of the color filters on the mosaic. 
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Once the high resolution still image 1 10 is demosaiced, the high resolution still 
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image 110 may be color corrected depending on the illumination present at the time of the 
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capture (block 432). Complex transformation matrixes may be involved to restore 
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1 accurate color to the high resolution still images 1 10, in order to generate an excellent 

2 photograph. The color correction algorithms, may be similar to the algorithm used in the 

3 HP-PHOTOSMART 618®. 

4 Figure 4C illustrates in detail the operation of the video pipeline 220. A high 

5 quality video pipeline 220 may be very demanding in terms of computation. Because the 

6 video processing needs to be achieved at, for example, 30 fps, downsampling may be fast. 

7 In addition, lower resolution video frames 120 (for example, 720x480 pixels) demands 

8 much less quality demosaicing (block 410), because the human visual system may not 

9 notice certain artifacts at high video processing rates. For example, demosaicing for 

10 video frames 120 may filter the original image with a 4x4 linear filter. Similarly, color 

1 1 correction may be simpler because high quality is not needed on the video side (block 
§ 12 430). 

2 13 Figure 5 illustrates an alternative embodiment of a video camera system 500 using 

Im 14 the exemplary concurrent dual video and still image pipeline. In this embodiment, two 

P 15 separate hardware processing pipelines may run concurrently within the video camera 

~ s 16 system 500, using two separate image sensors 540, 542. For the video processing 

s 

M ! 17 pipeline 220, a medium resolution sensor 540 (for example, 720x480 pixels) is used, 

III 

18 whereas a high resolution sensor 542 (for example, 2000x2000 pixels) is used for the high 

H 5 19 resolution still image pipeline 210. With separate hardware processing pipelines, no 

yk 20 interference exists, therefore no large frame buffer is needed. Even though presenting an 

21 easier solution, this alternative embodiment may be more expensive, because duplicate 

22 hardware are involved while the two hardware processing pipelines may be used 

23 simultaneously only occasionally. Alternatively, the two separate hardware processing 

24 pipelines may use the same image sensor (not shown). One skilled in the art will 

25 appreciate that many other intermediate solutions will be possible with this architecture 

26 for the concurrent dual video and still image pipeline. 

27 While the concurrent dual video and still image pipeline has been described in 

28 connection with exemplary embodiments, those skilled in the art will understand that 

29 many modifications in light of these teachings are possible, and this application is 

30 intended to cover any variation thereof. . .. 
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